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ABSTRACT 



Dilemmas emerged in the evaluation of a school improvement 
project that required evaluators to confront merging varied and often 
conflicting perspectives, sources of data, understandings of their roles as 
evaluators, the eventual use of the data, and the purpose of the educational 
reform effort. Selected examples are presented from the evaluation study in 
order to illuminate the challenges evaluators face when involved in 
multi-level, comprehensive studies of evolving school improvement projects. 
The dilemmas are presented in order to foster thoughtful discussions of the 
challenges, opportunities, and obligations evaluators face in their work with 
participants and those who receive their findings. Data were collected at 
five middle schools in the New Orleans (Louisiana ) area that were 
participating in the Learning Connections Project. Data were collected from 
each principal, 26 teachers, 461 students, and 21 parents. Linking the 
diverse data produced representation in determining weight in decision 
making. Data reduction by its nature entails losing information, and the 
challenges it poses can be viewed as negative consequences of program 
evaluation. However, they can also be seen as the impetus to reflective, 
careful, and thoughtful work. (Author/SLD) 
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Whose Voice Do You Hear When Participants’ Voices Clash? 

Abstract 

Dilemmas emerged throughout the evaluation of a school improvement project that 
required us to confront merging varied and often conflicting perspectives, sources of data, and 
understandings of our roles as evaluators, the eventual use of the data, and the purpose of the 
educational reform effort itself We present selected examples from our evaluation study in order 
to illuminate the challenges evaluators face when involved in multi-level, comprehensive studies 
of evolving school improvement projects. The dilemmas are presented in order to foster 
thoughtful discussion of the challenges, opportunities, and obligations evaluators face in their 
work with participants and those who receive their findings. 
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Whose Voice Do You Hear When Participants’ Voices Clash? 

Introduction 

By their very nature, educational reform evaluations involving mixed 
qualitative/quantitative evaluations pose particular dilemmas for evaluators in terms of analyzing 
data and communicating findings. The focus in this paper is on the challenges evaluators face in 
hearing and representing participants’ multiple and sometimes conflicting voices captured by 
diverse data sources. Our purpose is to illuminate some of the dilemmas inherent in merging 
various types of information, using the Learning Connections Project as the setting for exploring 
issues of voice in evaluation of comprehensive, multi-level school reform efforts. 

Background 

Tensions lie in the merging of varied forms of data as well as in melding varied and often 
conflicting perspectives of the same phenomenon. The following quotation highlights many of 
the challenges evaluators face when confronted with multiple sources of data: "The critical point 
is how to integrate these sources of data. Comparison of the different accounts is possible, but 
integration within a single narrative not only is difficult, but also obscures the very complexity 
one worked hard to document" (Fielding & Fielding, 1986, p. 26). Thus, evaluators are faced 
with dilemmas in combining diverse sources of information, with understanding and representing 
varied and sometimes divergent perspectives, and with faithfully representing dynamic, complex 
teaching and learning phenomena in succinct, accessible reporting in order to assess educational 
reform efforts. 

Researchers and evaluators must take into account the nature of the evaluator-participant 
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relationship and its effect on the information gained. From the perspective of the participants in 
the study, there are social dimensions of the relationship that affect their responses. Perret- 
Clement, Ferret, and Bell (1991) report that children in their study of conservation of liquids 
devoted significant effort to understanding the experimenter-subject relationship. They 
conducted a set of experiments that investigated the social interaction between experimenter and 
child, and found that children made specific and sustained attempts to understand the 
experimenter’s expectations of them, as well as trying to understand the cognitive demands of 
the task. Differences in performance were linked to a variety of parameters, including social 
class, race, gender, and the children’s prior experiences in experimental situations. As they point 
out, "Indeed, the cognitive activity of the subject applies not only to his or her understanding of 
the logical features of the task but also to the task’s meaning within its context and to the 
understanding of the social relationships that partners (experimenter or peers) establish around 
the task" (Perret-Clermont, Ferret, & Bell, 1991, p. 51). 

Rubin and Rubin (1995) provide a useful example of the many ways in which 
interviewees’ perceptions of the role of the interviewer influence their responses. In working 
with community development activists, one of the authors was treated as therapist, social change 
agent, scholar, and historian in the course of the research. The result was that "the roles into 
which Herb was cast affected what was said in the interviews. With a therapist, the interviewees 
could expose weaknesses and stresses. With a social activist, they could discuss strategies. To a 
historian of their movement, they could describe their triumphs in a world of adversity" (Rubin 
& Rubin, 1995, p. 115). 
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The Setting 

The Schools 

The data were collected at five middle schools in the New Orleans area that were 
participating in the Learning Connections Project. These represent a variety of school settings, 
including largely middle-class to low-income communities. As a group, they serve a fairly 
diverse student population in terms of ethnicity and socioeconomic status. The 5 schools also 
represent both urban, suburban, and semi-rural communities. Thus, the Learning Connections 
Project is operating in dynamic, complex settings that involve efforts at multiple levels of 
schooling. 

Participants 

Participants in the evaluation study included the principals of each middle school, 
randomly selected teachers (26) and students (461), and parents (21) who volunteered to be 
involved. The teachers represent a range of disciplines from history, science, and language arts to 
physical education and drug awareness. The range of classes, therefore, covers the many different 
settings in which students learn. All students in those classes and whose parents gave informed 
consent were included in videotaped class sessions. Students were randomly selected from that 
group to be interviewed individually or in focus groups. 

Instruments 

Observation materials . Initial observation materials involved open-ended note taking of 
all activities videotaped in each class that fell into three categories: variety of instruction, respect 
for diversity of learning, and student engagement. There also was room on the observation sheet 
for further comments about any other aspect of the classes that were being observed. This 
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intentionally-open record-keeping format was suited to the exploratory nature of the 
observations. 

In a later stage of data organization, we developed a ratings sheet that represented 
numerically the extent to which teachers were varying instruction and the extent to which 
students were engaged in learning in the classrooms. Each item on these sheets was rated on a 
scale from 1 to 5, where 1 represented the lowest level and 5 represented the highest level of 
teacher or student behavior in each category. These rating sheets grew directly out of our 
following a sequence of accepted data-summary steps, described in the next section, to lead to 
valid numerical summaries that supplement the qualitative descriptions. 

Interviews . Interview questions were developed for principals, teachers, students, and 
parents, based upon the explicit objectives of the Learning Connections Project. While these 
questions were developed to address objectives directly, they were semi-structured in that the 
persons being interviewed were encourage to expand on their responses or to address other topics 
if they desired. This method of interviewing kept open the possibility that important issues other 
than those anticipated in the evaluation plan might surface during the interviews. 

Learner-Centered Battery . The Learner-Centered Battery (LCB) is a paper-and-pencil 
survey developed by the Mid-Continent Region Educational Laboratories (McREL, 1 996) to 
explore the degree to which teachers perceive themselves as being learner-centered in their 
teaching practice and for students to assess the leamer-centeredness of their teachers’ classroom 
practices. This instrument provides information about teachers’ and students’ assessments of 
classroom practices; teacher beliefs about students, teaching and learning; and selected student 
learning variables. 
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School Practices Survey . The School Practices Survey (SPS; McREL, 1997) was 
developed to assess the degree to which key school personnel (e.g., principals and teachers) not 
only agree with certain learner-centered principles but also the degree to which they believe their 
school lives up to each of those principles. Participants rate each item twice (once concerning the 
importance of that principle and once concerning the school’s implementation of that principle) 
on 5-point scales that range from strongly agree to strongly disagree. 

The Dilemmas 

Two fundamental processes in qualitative research are triangulation and data reduction 
(Denzin, 1989; Fielding & Fielding, 1986; Huberman & Miles, 1998). They are in some sense 
competing processes. In triangulation, multiple data sources are gathered in order to explore 
several perspectives on a particular phenomenon, while in data reduction, rich, varied data are 
summarized into various forms, including themes, quotations, and frequency counts. In our case, 
we used several types of data (individual and focus group interviews, classroom observations, 
and surveys) and we drew on the experiences of several groups of people (students, parents, 
teachers, and principals) in order to understand the impact of the Learning Connections Project 
on teaching and learning. Reduction of data occurred at several stages in the first year of the 
Project: 1) during statistical and content analyses; 2) via ongoing, informal communication 
between evaluators and Project staff; 3) as part of producing informal written summaries of 
evaluators’ perceptions and initial understandings of the data; and 4) for production of a year-end 
formal report of our findings. Thus, dilemmas in whose voices we heard and how we represented 
those voices emerged throughout the year. Those dilemmas involve the ways in which the nature 
of the data we gathered influenced the voices we heard. They also involve participants’ and 
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evaluators’ understanding of the purpose of the project and the data collection procedures. 

The Nature of the Data Shapes the Voices 

Stakeholders . Our evaluation design grew from our work with Project staff in formulating 
objectives that matched their goals. The formulation of those objectives in and of itself shaped 
whose voices were prominent in that particular perspectives were represented (i.e., Project staff 
and evaluators) and others were not (i.e., principals, teachers, students, etc. whose teaching and 
learning were the focus of the reform efforts). Thus, some stakeholders had a voice in 
determining what was measured and investigated, while others did not. This is a traditional 
evaluation procedure, but some evaluators (Guba & Lincoln, 1989; Patton, 1994) suggest more 
inclusive processes of hearing stakeholders’ voices. For example, in the Learning Connections 
evaluation plan, we could have invited teachers, parents, principals, and students to participate in 
formulating the objectives and to help determine the types of evidence to gather of the impact of 
the Project. 

Data. We designed our evaluation in response to the Project objectives, choosing a mix of 
qualitative and quantitative approaches in order to capture the complexity of the effort. The semi- 
structured interviews involved open-ended questions and the expectation that the interviewer 
would ask follow-up questions, encourage elaboration, and allow those they were interviewing to 
expand on their answers. We videotaped class sessions so that we could revisit the setting and 
conduct holistic observations of interactions and activities. Surveys through which teachers and 
students assessed classroom practices and teachers and principals assessed school practices gave 
us another view of participants’ experiences. Thus, in data collection, we were interested in 
hearing from those we interviewed and observed in a variety of ways and at as deep a level as 
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was possible within an evaluation design that focused on measuring outcomes. 

While our design was one where we were consciously trying to include many voices, the 
interview and survey questions did shape what those voices talked about. Among the myriad 
questions we could ask in interviews, those we did pose elicited some important information, but 
the questions we did not ask left out information that may also be of importance to our 
understanding the impact of the Project in the schools. The surveys we chose to administer both 
investigate school and classroom practices from a particular theoretical framework (learner- 
centered principles), again shaping the topic about which the voices spoke. Thus, decisions made 
from the beginning of our involvement in the Project, through the design phase of the evaluation, 
and into the collection of data shaped the voices that contributed to our efforts to understand how 
change was occurring in the schools. 

Perceptions of Purposes of the Project, the Evaluation, and the Data 

Participants’ understanding of the purpose of the Project, the eventual use of the data, and 
our roles as evaluators also shaped the voices we heard. Each of the groups involved had their 
unique roles in the schools, roles that influenced their perceptions of the Project, of us as 
evaluators, and of the meaning of their participation. The examples presented next illustrate how 
the perceptions of particular groups involved in our evaluation shaped their responses. 

Perceptions of the purpose of the Project and of the evaluation . Principals, as leaders of 
their schools and having chosen Learning Connections as an impetus for improving teaching and 
learning, had a certain commitment to the Project. Their answers to interview questions were, of 
course, colored by what they thought they should be doing in the Project as a result of that 
commitment. Also, given the open-ended and semi-structured nature of the questions, many may 
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have seen them as evaluations of them rather than of the Project. Questions for principals (e.g., 

"How do you go about supporting effective instructional practices?" and "What do you do to 

encourage teachers’ increasing the variety of their instructional strategies") were meant by the 

interviewers to give us baseline information about current practices so that we could detect any 

changes later. The principals may well have interpreted the questions as evaluative rather that 

information-seeking. That interpretation would color their responses and their voices would be 

shaped by their interpretation of the purpose of the questions and of the evaluation. 

Also, the principals’ purpose for having their schools participate in the Learning 

Connections Project differed, as evidenced in the following quotations: 

CDL for me is a lot of staff development, . . . [Faculty could see] that CDL is the current philosophy or 

trend or whatever, having more people reinforcing that it’s better, that it’s not just from me, coming from 
home in California saying ‘Oh guess what, you don’t have to teach the book.’ 

And that school improvement plan has worked for us for the last few years, we’re a model middle school. 
And CDL fit right in with the kinds of things that we have been writing about and working with and 
dealing with now. Every thing is geared toward what is now a model middle school. So those kinds of 
things were already in the mill. We have just expanded a little. 

The differing purposes illustrated in the principals’ statements colored their responses to our 
questions. Taken together, principals’ perceptions of our roles as evaluators and their beliefs 
about the purposes of the Project in their schools were very influential in shaping what they said 
in interviews. 

Understanding of the Eventual Use of the Data. Teachers’ varying expectations about 
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what purposes the data would serve shaped their responses in interesting ways. Many teachers, 
especially early in the Project, were watchful, seemingly withholding judgment of the Project in 
order to determine whether or not it was another in the long line of "quick-fix"programs that they 
had seen come and go over the years. Some had concerns that the time they were spending 
working with us rather than on other aspects of their teaching might be wasted. Their 
reservations and skepticism shaped their responses to our interview questions in that they were 
guarded in their answers. An example of this was seen in one teacher’s response in which she 
said she faced no barriers to operating in the way she thought best in her classroom and that there 
were no students she was not reaching. Another perspective was evident in the responses from 
two novice teachers at one school whose principal was a strong, controlling leader. Although we 
assured them that the results of their interviews would remain confidential, interviewers came 
away from interactions with them with the impression that they were careful in their responses to 
express their support of her and her policies. When responding to our questions, other teachers 
saw an opportunity to receive feedback on their instructional practices from people they 
considered to be knowledgeable and spoke at length about areas in which they wanted input. 
Finally, some teachers took to heart our statements that the information we were gathering from 
them was to be used to give feedback to the Learning Connections staff so that they could tailor 
their efforts to fit teachers’ and schools’ needs. They formed their answers to questions with an 
eye to shaping the type of help they wanted to receive from the Project. These variations in 
perception and in response presented us with dilemmas in interpretation, when we were faced 
with merging the disparate voices. 

Data Analysis. Reduction and Reporting 
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While we were gathering, analyzing, and reporting the data, we encountered additional 
dilemmas concerning voice. In the process of analyzing the interviews and observations, the 
steps we went through in content analysis of course reduced and summarized what was very rich, 
deep data. Remaining true to the voices of the participants and representing their experiences and 
perspectives were challenges that we faced in our analyses and in reporting. We also confronted 
the fact that the voices we were hearing often presented conflicting views of what was happening 
in the classrooms and schools. That data reduction and our representation of the voices in our 
reporting further shaped the voices of the participants. 

Collective voice versus collections of individual voices . The mere fact that the numbers 
of people in each group in our evaluation differed influenced the voices we heard. There were 
five principals, one from each of the schools, 26 randomly selected teachers, 461 randomly 
selected students involved, as well as 21 parents who volunteered. In our analysis and reporting, 
the five individual principals became a "principals," the 26 individual teachers became 
"teachers," etc. Their individual voices became a collective voice. For example, these individuals 
were asked to comment in some way through interviews and surveys on the instructional 
practices of teachers and, in the case of teachers, on their own instructional practices. By the 
time of analysis, each group of participants became noted as that particular group (i.e., the 
students, the teachers, the principals, the parents). Even though there were five different 
principals with different ideas, perspectives, and issues, they were aggregated together 
collectively to report how all of the principals felt. This phenomenon happened with the other 
groups of participants and the schools, as well. The principals and teachers had individual 
interviews such that their information was personal and individual; their individual responses 
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were then combined with the others in the group to report the collective voice. Additional 
examples of individual voices becoming a collective voice further illustrate our dilemmas. 
Schools 

Each of the schools presents it own unique set of strengths and needs, so that our 
evaluation had to provide information for Learning Connections staff to help them tailor their 
efforts to fit each school. In formal reporting, however, the schools were treated as a single 
entity. For example, a substantial increase in the numbers of students and faculty is a challenge 
for one school, parent involvement and community support is a focus for another, and basic 
resources such as adequate bathrooms, air conditioning, and having enough textbooks are needs 
at another. On the other hand, all five schools are committed to enhancing student learning and 
achievement through attention to increasing instructional variety, addressing more effectively the 
diversity in learning that students present, and involving parents and community members in 
decision making. The challenge, then, was for our evaluation to inform Learning Connections 
staffs’ efforts to serve both the unique and the common interests of the five schools in which they 
were working. The built-in flexibility and the comprehensive nature of the Project were 
important strengths to be maintained as each school created its own path in reform, while still 
focusing on the shared challenges. 

Principals 

An example of the loss of rich data is illustrated by the following example. In the year- 
end report, principals were reported as being committed to increeising parent involvement 
and developing better relations with parents. However, each school starts at a different level of 
the playing field with respect to improvement of parental involvement and where their particular 
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parents are in terms of quality and scope of involvement. The types of activities principals 
reported as being available to parents varied considerably from school to school, and they 
involved both decision making and/or academic activities and non-academic activities. 

Responses from individual principals ranged from "parents are involved with strategic planning 
committees" to "parents are involved with Parent and Pizza Night." In terms of academics and 
decision making, two schools stood out from the others. At those schools, parents served on 
school improvement committees and/or revised school improvement plans. 

The collective voice represented here may be characterized in very different ways when 
compared to the conversations of the collective voice at other campuses. In these two schools, 
parental questions and input could involve the quality of instruction, academic resources, school 
organization, and curriculum issues. On the other hand, the principals at the other campuses 
might have begun the conversation with the collective voice of parents at a more basic level, a 
level which included parental attendance to school activities as opposed to participation, input, 
and decision making about school activities, in addition to the maintenance of school grounds. At 
those three schools, the collection of individual voices of the parents interviewed in focus groups 
reported involvement in non-academic activities which included such things as painting, 
maintenance, and remodeling, booster club for athletics and school bands, and helping in the 
school office. Hence, a central issue highlighted by principals’ interviews was that of gaining the 
legitimate participation of a large number and wide variety of parents. As evaluators, balancing 
the scale of how to represent a collective voice as opposed to a collection of individual voices 
remains an important challenge. 

Teachers 

For the year-end final report, the voices of teachers were organized into a discussion 
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around three primary topics: variety of instruction, attitudes towards students and learning, and 
professional development. In general, the evaluation determined that the teachers in the project 
were becoming more aware of the value of increasing the variety of instructional methods, 
increasing their perceptions that they can reach many students, and experiencing benefits from 
professional development. However, the collection of individual voices of teachers demonstrated 
some specific, perhaps more substantial information, about each of the three primary topics. In 
the area of student motivation, the teachers answers ranged from very motivated to not at all. 
Many of the teachers categorized the students’ motivation as being external. However, there are 
several different types of external motivation represented in their answers. One teacher reported 
that students "don’t see too far down the road. They see that their parents have done okay." 
Another teacher reported that students were held back, frustrated, and had experienced too many 
failures, which is also an external motivation. A third teacher reported that students who excel 
may experience negative peer pressure. Thus, a variety of detail was necessarily subsumed in the 
reporting of the data. 

Merging conflicting voices . Voices of individuals and of groups, captured by surveys, 
interviews, and observations, presented us with conflicting information and perspectives on 
numerous aspects of classrooms and schools. We faced dilemmas in merging those disparate 
voices as we sought to represent the complex, dynamic nature of the circumstances producing 
those conflicting perspectives. For example, teachers’ assessments of their classroom practices as 
measured by the Learner-Centered Battery indicated that they think they are fairly learner- 
centered in their approaches. The majority of the teachers also reported in interviews that they 
were modifying their practices based on increased awareness of learning differences. However, 
students’ assessment on the same survey and their responses in interviews indicated that they 
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experience their classrooms as more teacher-centered. Our analyses of classroom observations 
corroborate the students’ assessments. 

Merging those data and perspectives into a coherent picture of instructional practices 
required that we present the students’ and teachers’ distinct voices in such a way as to faithfully 
represent classroom interactions. Merging those disparate sources of information also required 
that we weigh the credibility of both the types of data (survey, observation, interview) and whose 
voices we were hearing. It was at this point that our own voices came very much into play in that 
we based our decisions, of course, partly on our own perspectives of teaching and learning and 
also on the fact that improving student learning and performance were the ultimate goals of the 
Project. 

In the particular case of assessing classroom practices, students’ perspectives were given 
considerable weight, based on research that shows that their experiences are more closely linked 
to their learning and performance than are teachers’ beliefs about their practices. Thus, linking 
diverse data together to form a picture of teaching and learning and to form the basis of our 
evaluation produced dilemmas both in representation and in determining weight in our decision- 
making activities. 

Conclusions and Discussion 

The dilemmas we present here are meant to promote thoughtful consideration of some of 
the challenges evaluators face in bringing together varied and sometimes divergent perspectives 
in order to represent faithfully and to assess the phenomena under study. Data reduction by its 
very nature entails losing information, including contextual information that provides a setting in 
which we can situate events, perspectives, and interactions. As presented in earlier sections of 
this paper, varied responses to interview questions that give a sense of the complexity of teaching 
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and learning are sometimes reduced to single numbers. Along with the loss of context, the 
richness of the voices can suffer as well. Important voices can be silenced if their numbers are 
low compared to others in the sample. Accountability to stated objectives can prevent recognition 
of unexpected outcomes and processes that may be crucial to sustained, positive change. While 
those challenges can be viewed as negative consequences of the process of evaluating the impact 
of programs such as the Learning Connections Project, they can also be approached as impetus to 
be reflective, careful, and thoughtful in our work. They are reminders of the obligations 
evaluators have to those they study and to those who receive our findings. 
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